How to Identify an Unexpected Shutdown of a CUCM, UC, UCCX, CER, or Cisco Prime Server

Introduction

This document describes how to identify an unexpected shutdown of a Cisco Unified Communications Manager (CallManager) (CUCM), Cisco Unity Connection (UC), Cisco Unified Contact Center Express (UCCX), Cisco Emergency Responder (CER), Cisco Prime, or any application that runs on top of Cisco’s customized Voice Operating System (VOS). If the server experiences an unexpected shutdown, the file system consistency cannot be guaranteed. Files might be removed unexpectedly, ownership of file permissions might be changed, or the contents of files might be corrupted.

In order to temporarily recover the system, run the system recovery disc released for the corresponding software version.

Verify Improper Shutdown

Review the system-history.log in order to determine if a system has been shut down improperly.

Note: The system-history.log was added as part of Cisco bug ID CSCsl94283, “CCM 5.X should track all install/upgrade with History.log like 4.X.” Versions in earlier releases were not tracked. The history.log was enhanced in order to track improper shutdowns with Cisco bug ID CSCtr88859 in order to add alarms and alerts for unexpected reboots that are integrated in CUCM Versions 9.1(1) and later.

1. Download the install/upgrade logs from the Cisco Unified Real-Time Monitoring Tool (RTMT), and gather the system-history.log.
   or
   Enter the file view install system-history.log command on the command-line interface (CLI).
2. Examine each instance of root: Boot, and confirm that each instance is preceded by one of these lines:

   root: Restart
   root: Shutdown
   root: Install
   root: Upgrade

3. If a boot instance is not proceeded by a Restart, Shutdown, Install, or Upgrade, there was likely an unclean shutdown.

This is an example of an unclean shutdown:

08/14/2012 13:36:09 | root: Boot 9.0.1.10000-37 Start
08/14/2012 17:28:25 | root: Boot 9.0.1.10000-37 Start

In this example, the server must be rebuilt in order to ensure file system consistency. See these Cisco bug IDs for further details:

• Cisco bug ID CSCth60800, “Recovery Disc warning to rebuild system after file system repair”
• Cisco bug ID CSCth53322, “Document the need for system rebuild after file system repair”
• Cisco bug ID CSCuy94644, “Cisco Emergency Responder corruption after unexpected shutdown”

Note: If the server runs on VMware on a version without the fix for Cisco bug ID CSCtw73590, “VSphere initiated shutdown or restart not logged to system-history.log” and if the server is shut down through VSphere when a guest shutdown is initiated, that entry might not be included in the system-history.log.

http://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/116717-trouble-cucm-shutdown-00.html